{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [ Chapter 3 - Ranking and Content-based Relevance ] \n",
    "# Vectors and Text Similarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy\n",
    "from aips import num2str, tokenize, vec2str\n",
    "from numpy.linalg import norm\n",
    "from numpy import dot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Listing 3.1\n",
    "In this example, we explore Ranking two documents for the query \"Apple Juice\". We present the query as a feature vector, as well as the documents.\n",
    "\n",
    "#### Text Content\n",
    "*Query*: \"Apple Juice\"\n",
    "\n",
    "*Document 1*: \n",
    "```Lynn: ham and cheese sandwich, chocolate cookie, ice water.\n",
    "Brian: turkey avocado sandwich, plain potato chips, apple juice\n",
    "Mohammed: grilled chicken salad, fruit cup, lemonade```\n",
    "\n",
    "*Document 2*: ```Orchard Farms apple juice is premium, organic apple juice made from the freshest apples, never from concentrate. Its juice has received the regional award for best apple juice three years in a row.```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Dense Vectors\n",
    "If we consider a vector with each keyword as a feature (48 terms total):\n",
    "```[a, and, apple, apples, avocado, award, best, brian, cheese, chicken, chips, chocolate, concentrate, cookie, cup, farms, for, freshest, from, fruit, grilled, ham, has, ice, in, is, its, juice, lemonade, lynn, made, mohammed, never, orchard, organic, plain, potato, premium, received, regional, row, salad, sandwich, the, three, turkey, water, years]```\n",
    "\n",
    "\n",
    "Then our query becomes the 48-feature vector, where the `apple` and `juice` features both exist:\n",
    "Query:      ```[0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]```\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "query_vector = numpy.array(\n",
    "    [0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
    "     0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The corresponding vectors for our documents are as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = 1\n",
    "b = a\n",
    "a += b * a "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc1_vector = numpy.array(\n",
    "    [0, 1, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 1,\n",
    "     0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 0])\n",
    "doc2_vector = numpy.array(\n",
    "    [1, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0,\n",
    "     1, 1, 1, 1, 0, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Similarity\n",
    "To rank our documents, we then just need to calculate the cosine between each document and the query, \n",
    "which will become the relevance score for each document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cosine_similarity(vector1, vector2):\n",
    "    return dot(vector1, vector2) / (norm(vector1) * norm(vector2))\n",
    "\n",
    "def print_scores(scores):\n",
    "    print(\"Relevance Scores:\")\n",
    "    for i, score in enumerate(scores):\n",
    "        print(f\"  doc{i + 1}: {num2str(score)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Relevance Scores:\n",
      "  doc1: 0.2828\n",
      "  doc2: 0.2828\n"
     ]
    }
   ],
   "source": [
    "doc1_score = cosine_similarity(query_vector, doc1_vector)\n",
    "doc2_score = cosine_similarity(query_vector, doc2_vector)\n",
    "\n",
    "print_scores([doc1_score, doc2_score])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Interesting... Both documents received exactly the same relevance score, even though the documents contain lengthy vectors with very different content. It might not be immediately obvious, but let's simplify the calculation by focusing only on the features that matter.\n",
    "\n",
    "#### Sparse Vectors\n",
    "The key to understanding the calculation is understanding that the only features that matter are the ones shared between the query and a document. All other features (words appearing in documents that don't match the query) have zero impact on whether one document is ranked higher than another. As such, we can simplify our calculations significantly by creating sparse vectors that only include the terms present in the query."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Listing 3.2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Relevance Scores:\n",
      "  doc1: 1.0\n",
      "  doc2: 1.0\n"
     ]
    }
   ],
   "source": [
    "query_vector = [1, 1] #[apple, juice]\n",
    "doc1_vector  = [1, 1]\n",
    "doc2_vector  = [1, 1]\n",
    "\n",
    "doc1_score = cosine_similarity(query_vector, doc1_vector)\n",
    "doc2_score = cosine_similarity(query_vector, doc2_vector)\n",
    "\n",
    "print_scores([doc1_score, doc2_score])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In fact, you'll notice several very interesting things:\n",
    "1. This simplified sparse vector calculation still shows both `doc1` and `doc2` returning equivalent relevance scores, since they both match all the words in the query.\n",
    "2. Even though the absolute score between the dense vector similarity (0.2828) and the sparse vector similarity (1.0) are different due to normalization, the scores are still the same relative to each other (equal to each other in this case).\n",
    "3. The feature weights for the two query terms (`apple`, `juice`) are exactly the same between the query and each of the documents, resulting in a cosine score of 1.0.\n",
    "\n",
    "The problem here, of course, is that the features in the vector only signifies IF the word `apple` or `juice` exists, not how well each document actually represents either of the terms. We'll correct for by introducing the concept of \"term frequency\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Term Frequency\n",
    "In Section 3.1.4 we learn about Term Frequency (TF). If we count up the number of times each term appears in our documents, we will get a better understanding of \"how well\" the document represents those terms:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "query_term_occurrences: [1, 1]\n",
      "doc1_term_occurrences: [1, 1]\n",
      "doc2_term_occurrences: [3, 4]\n"
     ]
    }
   ],
   "source": [
    "query = \"apple juice\"\n",
    "doc1 = \"Lynn: ham and cheese sandwhich, chocolate cookie, ice water.\\nBrian: turkey avocado sandwhich, plain potato chips, apple juice\\nMohammed: grilled chicken salad, fruit cup, lemonade\"\n",
    "doc2 = \"Orchard Farms apple juice is premium, organic apple juice  made from the freshest apples and never from concentrate. Its juice has received the regional award for best apple juice three years in a row.\"\n",
    "\n",
    "query_term_occurrences = [tokenize(query).count(\"apple\"), tokenize(query).count(\"juice\")] #[apple:1, juice:1]\n",
    "doc1_term_occurrences = [tokenize(doc1).count(\"apple\"), tokenize(doc1).count(\"juice\")] #[apple:1, juice:1]\n",
    "doc2_term_occurrences = [tokenize(doc2).count(\"apple\"), tokenize(doc2).count(\"juice\")] #[apple:3, juice:4]\n",
    "\n",
    "print(f\"query_term_occurrences: {query_term_occurrences}\")\n",
    "print(f\"doc1_term_occurrences: {doc1_term_occurrences}\")\n",
    "print(f\"doc2_term_occurrences: {doc2_term_occurrences}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the feature values for the terms `apple` and `juice` are now weighted in each vector based upon the number of occurrences in each document. Unfortunately we can't just do a cosine similarity on the raw count of term occurrences, however, because the query only contains one occurrence of each term, whereas we would consider documents with multiple occurrences of each term to likely be more similar. Let's test it out and see the problem:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Listing 3.3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Relevance Scores:\n",
      "  doc1: 1.0\n",
      "  doc2: 0.9899\n"
     ]
    }
   ],
   "source": [
    "query_vector   = [1, 1] #[apple:1, juice:1]\n",
    "doc1_tf_vector = [1, 1] #[apple:1, juice:1]\n",
    "doc2_tf_vector = [3, 4] #[apple:3, juice:4]\n",
    "\n",
    "doc1_score = cosine_similarity(query_vector, doc1_tf_vector)\n",
    "doc2_score = cosine_similarity(query_vector, doc2_tf_vector)\n",
    "\n",
    "print_scores([doc1_score, doc2_score])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since our goal is for documents like `doc2` with higher term frequency to score higher, we can overcome these by either using switching from cosine similarity to another scoring function, such as _dot product_ or _Euclidean distance_, that increases as feature weights continue to increase. Let's switch to using the dot product (`a . b`), which is equal to the cosine similarity multiplied by the length of the query vector times the length of the document vector: `a · b = |a| × |b| × cos(θ)`. The dot product will result in documents that contain more matching terms scoring higher, as opposed to cosine similarity, which scores documents higher containing a more similar proportion of matching terms between the query and documents. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Listing 3.4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_dot_product_scores():\n",
    "    print(f\"\"\"Relevance Scores:\n",
    " doc1: {num2str(doc1_score)}\n",
    " doc2: {num2str(doc2_score)}\"\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Relevance Scores:\n",
      "  doc1: 2\n",
      "  doc2: 7\n"
     ]
    }
   ],
   "source": [
    "query_vector   = [1, 1] #[apple:1, juice:1]\n",
    "doc1_tf_vector = [1, 1] #[apple:1, juice:1]\n",
    "doc2_tf_vector = [3, 4] #[apple:3, juice:4]\n",
    "\n",
    "doc1_score = dot(query_vector, doc1_tf_vector)\n",
    "doc2_score = dot(query_vector, doc2_tf_vector)\n",
    "\n",
    "print_scores([doc1_score, doc2_score])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result rankings now look more inline with our expectations. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great - the result rankings now look more inline with our expectations (for this simple example, at least)!\n",
    "\n",
    "As our feature-weighting calculation are getting more sophisticated, let's move on beyond our initial `\"apple juice\"` example toward a query and documents with more interesting statistics in terms of intersections and overlaps between terms in the query and terms in the documents.\n",
    "\n",
    "The following example demonstrates some more useful characteristics that will better help us understand how term frequency helps with our text-based sparse vector similarity scoring.\n",
    "\n",
    "*Document 1:* ```In light of the big reveal in her interview, the interesting thing is that the person in the wrong probably made a good decision in the end.```\n",
    "\n",
    "*Document 2:* ```My favorite book is the cat in the hat, which is about a crazy cat in a hat who breaks into a house and creates the craziest afternoon for two kids.```\n",
    "\n",
    "*Document 3:* ```My careless neighbors apparently let a stray cat stay in their garage unsupervised, which resulted in my favorite hat that I let them borrow being ruined.```\n",
    "\n",
    "Let's map these into their corresponding (sparse) vector representations and calculate a similarity score:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Listing 3.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc1 = \"In light of the big reveal in her interview, the interesting thing is that the person in the wrong probably made a good decision in the end.\"\n",
    "doc2 = \"My favorite book is the cat in the hat, which is about a crazy cat in a hat who breaks into a house and creates the craziest afternoon for two kids.\"\n",
    "doc3 = \"My careless neighbors apparently let a stray cat stay in their garage unsupervised, which resulted in my favorite hat that I let them borrow being ruined.\"\n",
    "docs = [doc1, doc2, doc3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_term_count_scores(terms, vectors, scores):\n",
    "    print(\"\\nlabels: \", terms)\n",
    "    print(f\"\\nquery vector: [{', '.join(map(num2str,query_vector))}]\\n\")\n",
    "    print(\"Document Vectors:\")\n",
    "    for i, doc in enumerate(vectors):\n",
    "        print(f\"  doc{i + 1}: [{', '.join(map(num2str,doc))}]\")\n",
    "    print(\"\\nRelevance Scores:\")\n",
    "    for i, score in enumerate(scores):\n",
    "        print(f\"  doc{i + 1}: {num2str(score)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def term_count(content, term):\n",
    "    tokenized_content = tokenize(content)\n",
    "    term_count = tokenized_content.count(term.lower())\n",
    "    return float(term_count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "labels:  ['the', 'cat', 'in', 'the', 'hat']\n",
      "\n",
      "query vector: [1, 1, 1, 1, 1]\n",
      "\n",
      "Document Vectors:\n",
      "  doc1: [5.0, 0.0, 4.0, 5.0, 0.0]\n",
      "  doc2: [3.0, 2.0, 2.0, 3.0, 2.0]\n",
      "  doc3: [0.0, 1.0, 2.0, 0.0, 1.0]\n",
      "\n",
      "Relevance Scores:\n",
      "  doc1: 14.0\n",
      "  doc2: 12.0\n",
      "  doc3: 4.0\n"
     ]
    }
   ],
   "source": [
    "query = \"the cat in the hat\"\n",
    "terms = tokenize(query)\n",
    "\n",
    "query_vector = list(numpy.repeat(1, len(terms)))\n",
    "doc_vectors = [[term_count(doc, term) for term in terms] for doc in docs]\n",
    "doc_scores = [dot(dv, query_vector) for dv in doc_vectors]\n",
    "\n",
    "print_term_count_scores(terms, doc_vectors, doc_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unfortunately, those results don't necessarily match our intuition about which documents are the best matches. Intuitively, we would instead expect the following ordering:\n",
    "1. doc2 (is about the book _The Cat in the Hat_ )\n",
    "2. doc3 (matches all of the words `the`, `cat`, `in`, and `hat`\n",
    "3. doc1 (only matches the words `the` and `in`, even though it contains them many times).\n",
    "\n",
    "The problem here, of course, is that since every occurrence of any word is considered just as important, the more times ANY term appears, the more relevant that document becomes. In this case, *doc1* is getting the highest score, because it contains 12 total term matches (`the` ten times, `in` two times), which more total term matches than any other document.\n",
    "\n",
    "To overcome these issues, \"term frequency\" calculations will typically both normalize for document length (take the total term count divided by document length) and also dampen the effect of additional term occurrences (take the square root of term occurrences).\n",
    "\n",
    "This gives us the following term frequency calculations:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Listing 3.6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_term_frequency_scores(terms, vectors, scores):\n",
    "    print(\"Document TF Vector Values:\")\n",
    "    for i, doc in enumerate(vectors):\n",
    "        print(f\" doc{i + 1}: [\" + ', '.join(map(lambda t : f'tf(doc{i + 1}, \"{t}\")', terms))+ \"]\")\n",
    "    print(\"\\nLabels:\", terms)\n",
    "    for i, doc in enumerate(vectors):\n",
    "        print(f\"  doc{i+1}: [{', '.join(map(num2str,doc))}]\")\n",
    "    print(\"\\nRelevance Scores:\")\n",
    "    for i, score in enumerate(scores):\n",
    "        print(f\"  doc{i+1}: {num2str(score)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With our updated TF calculation in place, let's calculate our relevance ranking again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Document TF Vector Values:\n",
      " doc1: [tf(doc1, \"the\"), tf(doc1, \"cat\"), tf(doc1, \"in\"), tf(doc1, \"the\"), tf(doc1, \"hat\")]\n",
      " doc2: [tf(doc2, \"the\"), tf(doc2, \"cat\"), tf(doc2, \"in\"), tf(doc2, \"the\"), tf(doc2, \"hat\")]\n",
      " doc3: [tf(doc3, \"the\"), tf(doc3, \"cat\"), tf(doc3, \"in\"), tf(doc3, \"the\"), tf(doc3, \"hat\")]\n",
      "\n",
      "Labels: ['the', 'cat', 'in', 'the', 'hat']\n",
      "  doc1: [0.4303, 0.0, 0.3849, 0.4303, 0.0]\n",
      "  doc2: [0.3111, 0.254, 0.254, 0.3111, 0.254]\n",
      "  doc3: [0.0, 0.1961, 0.2774, 0.0, 0.1961]\n",
      "\n",
      "Relevance Scores:\n",
      "  doc1: 1.2456\n",
      "  doc2: 1.3842\n",
      "  doc3: 0.6696\n"
     ]
    }
   ],
   "source": [
    "def tf(term, doc):\n",
    "    tokenized_doc = tokenize(doc)\n",
    "    term_count = tokenized_doc.count(term.lower())\n",
    "    doc_length = len(tokenized_doc)\n",
    "    return numpy.sqrt(term_count / doc_length)\n",
    "\n",
    "query = \"the cat in the hat\"\n",
    "terms = tokenize(query)\n",
    "\n",
    "query_vector = list(numpy.repeat(1, len(terms)))\n",
    "doc_vectors = [[tf(term, doc) for term in terms] for doc in docs]\n",
    "doc_scores = [dot(dv, query_vector) for dv in doc_vectors]\n",
    "\n",
    "print_term_frequency_scores(terms, doc_vectors, doc_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The normalized TF clearly helped, as `doc2` is now ranked the highest, as we would expect. This is mostly because of the dampening effect on number of term occurrences so that each additional term (in `doc1`, which matched `the` and `in` so many times) so that each additional occurrrence contributed less to the feature weight than prior occurrences. Unfortunately, `doc1` is still ranked second highest, so even that wasn't enough to get the better matching `doc3` to the top.\n",
    "\n",
    "Your intuition is probably screaming right \"Yeah, but nobody really cares about the words `the` and `in`. It's obvious that the words `cat` and `hat` should be given the most weight here!\"\n",
    "\n",
    "And you would be right. Let's modify our scoring calculation to fix this.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inverse Document Frequency (IDF)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Document Frequency (DF)* for a term is defined as the total number of document in the search engine that contain the term, and it serve as a good measure for how important a term is. The intuition here is that more specific or rare words (like `cat` and `hat`) tend to be more important than more common words (like `the` and `in`).\n",
    "\n",
    "$$DF(t\\ in\\ d)=\\sum_{d\\ in\\ c} d.contains(t)\\ ?\\ 1\\ :\\ 0$$\n",
    "\n",
    "Since we would like words which are more important to get a higher score, we take an inverse of the document frequency (IDF), typically defined through the following function:\n",
    "\n",
    "$$IDF(t\\ in\\ d)=1 + log (\\ totalDocs\\ /\\ (\\ DF(t)\\ +\\ 1\\ )\\ )$$\n",
    "\n",
    "In our query for `the cat in the hat`, a vector of IDFs would thus look as follows:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Listing 3.7"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_inverse_document_frequency_scores(terms, vector):\n",
    "    print(\"IDF Vector Values:\")\n",
    "    print(\"  [\" + ', '.join(map(lambda t: f'idf(\"{t}\")', terms)) + \"]\\n\")\n",
    "    #print(\"Labels: \", terms)\n",
    "    print(f\"IDF Vector:\\n  {vec2str(vector)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "IDF Vector Values:\n",
      "  [idf(\"the\"), idf(\"cat\"), idf(\"in\"), idf(\"the\"), idf(\"hat\")]\n",
      "\n",
      "IDF Vector:\n",
      "  [1.0512, 5.5952, 1.1052, 1.0512, 6.2785]\n"
     ]
    }
   ],
   "source": [
    "def idf(term):\n",
    "    #Mocked document counts from an inverted index\n",
    "    df_map = {\"the\": 9500, \"cat\": 100, \n",
    "              \"in\":9000, \"hat\":50}\n",
    "    totalDocs = 10000\n",
    "    return 1 + numpy.log(totalDocs / (df_map[term] + 1))\n",
    "\n",
    "terms = [\"the\", \"cat\", \"in\", \"the\", \"hat\"]\n",
    "idf_vector = [idf(term) for term in terms]\n",
    "\n",
    "print_inverse_document_frequency_scores(terms, idf_vector)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TF-IDF\n",
    "We now have the two principle components of text-based relevance ranking:\n",
    "- TF (measures how well a term describes a document)\n",
    "- IDF (measures how important each term is)\n",
    "\n",
    "Most search engines, and many other data science applications, leverage a combination of each of these factors as the basis for textual similarity scoring, using the following function:\n",
    "\n",
    "$$TF\\_IDF = TF * IDF^2$$\n",
    "\n",
    "With this formula in place, we can finally calculate a relevance score (that weights both number of occurrences and usefulness of terms) for how well each of our documents match our query:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Listing 3.8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_tf_idf_scores(terms, vectors, scores):\n",
    "    print(\"Document TF-IDF Vector Calculations\")\n",
    "    for i, doc in enumerate(vectors):\n",
    "        print(f\" doc{i + 1}: [\" + ', '.join(map(lambda t:\n",
    "            f'tf_idf(doc{i + 1}, \"{t}\")',terms)) + \"]\")\n",
    "\n",
    "    print(\"\\nDocument TF-IDF Vector Scores\")\n",
    "    print (\"Labels:\", terms)\n",
    "    for i, doc in enumerate(vectors):\n",
    "        print(f\"  doc{i + 1}: [{', '.join(map(num2str, doc))}]\")\n",
    "\n",
    "    print(\"\\nRelevance Scores:\")\n",
    "    for i, score in enumerate(scores):\n",
    "        print(f\" doc{i + 1}: {', '.join(map(num2str, score))}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tf_idf(term, doc):\n",
    "    return tf(term, doc) * idf(term)**2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Document TF-IDF Vector Calculations\n",
      " doc1: [tf_idf(doc1, \"the\"), tf_idf(doc1, \"cat\"), tf_idf(doc1, \"in\"), tf_idf(doc1, \"the\"), tf_idf(doc1, \"hat\")]\n",
      " doc2: [tf_idf(doc2, \"the\"), tf_idf(doc2, \"cat\"), tf_idf(doc2, \"in\"), tf_idf(doc2, \"the\"), tf_idf(doc2, \"hat\")]\n",
      " doc3: [tf_idf(doc3, \"the\"), tf_idf(doc3, \"cat\"), tf_idf(doc3, \"in\"), tf_idf(doc3, \"the\"), tf_idf(doc3, \"hat\")]\n",
      "\n",
      "Document TF-IDF Vector Scores\n",
      "Labels: ['the', 'cat', 'in', 'the', 'hat']\n",
      "  doc1: [0.4755, 0.0, 0.4702, 0.4755, 0.0]\n",
      "  doc2: [0.3437, 7.9519, 0.3103, 0.3437, 10.0126]\n",
      "  doc3: [0.0, 6.1397, 0.3388, 0.0, 7.7308]\n",
      "\n",
      "Relevance Scores:\n",
      " doc1: 1.4212\n",
      " doc2: 18.9623\n",
      " doc3: 14.2094\n"
     ]
    }
   ],
   "source": [
    "query = \"the cat in the hat\"\n",
    "terms = tokenize(query)\n",
    "\n",
    "query_vector = list(numpy.repeat(1, len(terms)))\n",
    "doc_vectors = [[tf_idf(term, doc) for term in terms] for doc in docs]\n",
    "doc_scores = [[dot(query_vector, dv)] for dv in doc_vectors]\n",
    "\n",
    "print_tf_idf_scores(terms, doc_vectors, doc_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finally!\n",
    "Finally our search results make intuitive sense! `doc2` gets the highest score, since it matches the most important words the most, followed by `doc3`, which contains all the words, but not as many times, followed by `doc1`, which only contains an abundance of insignificant words.\n",
    "\n",
    "This TF-IDF calculation is at the heart of many search engine relevance calculations, including the default algorithms - called BM25 - used by both Apache Solr and Opensearch. In addition, it is possible to match on much more than just text keywords - modern search engines enable dynamically specifying boosts of fields, terms, and functions, which enables full control over the relevance scoring calculation.\n",
    "\n",
    "We'll introduce each of these in the next workbook: [Controlling Relevance](2.controlling-relevance.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
